This invention relates to methods and arrangements for recognizing particular individuals occupying the passenger compartment of a vehicle and controlling vehicular components, including ignition of the vehicle, based on the recognition of the individuals.
1. Prior Art on Out of Position Occupants and Rear Facing Child Seats
Whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and over 100 people have now been killed. Thus, significant improvements need to be made to airbag systems. As discussed in detail in U.S. Pat. No. 5,653,462 referenced above, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. xe2x80x9cHow Airbags Workxe2x80x9d presented at the International Conference on Seatbelts and Airbags in 1993, in Canada, occupant position sensing and rear facing child seat detection systems are required.
Initially, these systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded airbag deployments when a front seat is unoccupied. However, airbags are now under development to protect rear seat occupants in vehicle crashes and all occupants in side impacts. A system will therefore be needed to detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782 will control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. As discussed more fully below, the instant invention is an improvement on that VIMS system and uses an advanced optical system comprising one or more CCD (charge coupled device) or CMOS arrays and particularly active pixel arrays plus a source of illumination preferably combined with a trained neural network pattern recognition system.
Others have observed the need for an occupant out-of-position sensor and several methods have been disclosed in U.S. patents for determining the position of an occupant of a motor vehicle. Each of these systems, however, has significant limitations. For example, in White et al. (U.S. Pat. No. 5,071,160), a single acoustic sensor and detector is described and, as illustrated, is mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed, by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. The sensors measure the change in position of an occupant during a crash and use that information to access the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in one or more of the above-referenced patents and patent applications. It is interesting to note that nowhere does Mattes et al. discuss how to use active or passive infrared to determine the position of the occupant. As pointed out in one or more of the above-referenced patents and patent applications, direct occupant position measurement based on passive infrared is probably not possible and, until very recently, was very difficult and expensive with active infrared requiring the modulation of an expensive GaAs infrared laser. Since there is no mention of these problems, the method of use contemplated by Mattes et al. must be similar to the electric eye concept where position is measured indirectly as the occupant passes by a plurality of longitudinally spaced-apart sensors.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant relative to the airbag since it is the impact of either the head or chest with the deploying airbag which can result in serious injuries. Both White et al. and Mattes et al. describe only lower mounting locations of their sensors in front of the occupant such as on the dashboard or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant""s hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry which ignores readings from some sensors if such readings are inconsistent with others, for the case, for example, where the driver""s arms are the closest objects to two of the sensors.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant""s hands as in the case where he/she is adjusting the knob on the radio and the remainder of the occupant. Three ultrasonic sensors of the to disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicated that the occupant was not moving while the third was indicating that he or she was. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they were blocking a substantial view of the occupant""s head or chest. Since the sizes and driving positions of occupants are extremely varied, it is now believed that pattern recognition systems and preferably trained pattern recognition systems, such as neural networks, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to suppress deployment if the occupant is out-of-position. In fact, the closer the occupant gets to the airbag, the faster the inflation rate of the airbag is according to the Fujita et al. patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat (occupant height). This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant""s head.
As discussed above, the optical systems described herein are also applicable for many other sensing applications both inside and outside of the vehicle compartment such as for sensing crashes before they occur as described in U.S. Pat. No. 5,829,782, for a smart headlight adjustment system and for a blind spot monitor (also disclosed in U.S. provisional patent application Ser. No. 60/202,424).
2. Definitions
Preferred embodiments of the invention are described below and unless specifically noted, it is the applicants intention that the words and phrases in the specification and claims be given the ordinary and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicant intends any other meaning, he will specifically state he is applying a special meaning to a word or phrase.
Likewise, applicants"" use of the word xe2x80x9cfunctionxe2x80x9d here is not intended to indicate that the applicants seek to invoke the special provisions of 35 U.S.C. xc2xa7112, sixth paragraph, to define their invention. To the contrary, if applicants wish to invoke the provisions of 35 U.S.C.xc2xa7112, sixth paragraph, to define their invention, they will specifically set forth in the claims the phrases xe2x80x9cmeans forxe2x80x9d or xe2x80x9cstep forxe2x80x9d and a function, without also reciting in that phrase any structure, material or act in support of the function. Moreover, even if applicants invoke the provisions of 35 U.S.C. xc2xa7112, sixth paragraph, to define their invention, it is the applicants"" intention that their inventions not be limited to the specific structure, material or acts that are described in the preferred embodiments herein. Rather, if applicants claim their inventions by specifically invoking the provisions of 35 U.S.C. xc2xa7 112, sixth paragraph, it is nonetheless their intention to cover and include any and all structure, materials or acts that perform the claimed function, along with any and all known or later developed equivalent structures, materials or acts for performing the claimed function.
The use of pattern recognition is central to the instant invention as well as to one or more of those disclosed in the above-referenced patents and patent applications above. xe2x80x9cPattern recognitionxe2x80x9d as used herein will generally mean any system which processes a signal that is generated by an object, or is modified by interacting with an object, in order to determine which one of a set of classes that the object belongs to. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally electrical signals coming from transducers which are sensitive to either acoustic or electromagnetic radiation and, if electromagnetic, they can be either visible light, infrared, ultraviolet or radar or low frequency radiation as used in capacitive sensing systems.
A trainable or a trained pattern recognition system as used herein means a pattern recognition system which is taught various patterns by subjecting the system to a variety of examples. The most successful such system is the neural network. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, sensor fusion, Kalman filters, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
A pattern recognition algorithm will thus generally mean an algorithm applying or obtained using any type of pattern recognition system, e.g., a neural network, sensor fusion, fuzzy logic, etc.
To xe2x80x9cidentifyxe2x80x9d as used herein will usually mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
To xe2x80x9cascertain the identity ofxe2x80x9d as used herein with reference to an object will generally mean to determine the type or nature of the object (obtain information as to what the object is), i.e., that the object is an adult, an occupied rear facing child seat, an occupied front facing child seat, an unoccupied rear facing child seat, an unoccupied front facing child seat, a child, a dog, a bag of groceries, etc.
An xe2x80x9coccupying itemxe2x80x9d or xe2x80x9coccupantxe2x80x9d of a seat or xe2x80x9cobjectxe2x80x9d in a seat may be a living occupant such as a human being or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
An optical image will generally mean any type of image obtained using electromagnetic radiation including infrared radiation.
In the description herein on anticipatory sensing, the term xe2x80x9capproachingxe2x80x9d when used in connection with the mention of an object or vehicle approaching another will usually mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact xe2x80x9cwith a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The targetxe2x80x9d vehicle is the vehicle that is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
xe2x80x9cOut-of-positionxe2x80x9d as used for an occupant will generally mean that the occupant, either the driver or a passenger, is sufficiently close to an occupant protection apparatus (airbag) prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. It may also mean that the occupant is not positioned appropriately in order to attain the beneficial, restraining effects of the deployment of the airbag. As for the occupant being too close to the airbag, this typically occurs when the occupant""s head or chest is closer than some distance such as about 5 inches from the deployment door of the airbag module. The actual distance where airbag deployment should be suppressed depends on the design of the airbag module and is typically farther for the passenger airbag than for the driver airbag.
3. Pattern Recognition Prior Art
Japanese Patent No. 3-42337 (A) to Ueno discloses a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses natural light and analyzes only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver""s mouth and eyes.
U.S. Pat. No. 5,298,732 to Chen also concentrates on locating the eves of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver""s eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes. Chen references the use of automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, there is no mention of illumination of the occupant, monitoring the position of the occupant, other that the eyes, determining the position of the eyes relative to the passenger compartment, or identifying any other object in the vehicle other than the driver""s eves. Also, there is no mention of the use of a trainable pattern recognition system.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant utilizing two spaced apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver""s eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. A review of this book did not appear to provide the answer to this question. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with active infrared radiation or in general electromagnetic radiation.
The use of neural networks as the pattern recognition technology is central to several of the implementations of this invention since it makes the monitoring system robust, reliable and practical. The resulting algorithm created by the neural network program is usually only a few dozen lines of code written in the C or C++ computer language as opposed to typically hundreds of lines when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the CCD and CMOS arrays, for example, have been prohibitively expensive until recently, rendering their use for VIMS impractical. Similarly, the implementation of the techniques of the above referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than $10 in large quantities.
The present invention preferably uses sophisticated trainable pattern recognition capabilities such as neural networks. Usually the data is preprocessed, as discussed below, using various feature extraction techniques. An example of such a pattern recognition system using neural networks on sonar signals is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. xe2x80x9cAnalysis of Hidden Units in a Layered Network Trained to Classify Sonar Targetsxe2x80x9d, Nueural Networks, Vol. 1. pp. 75-89, 1988, and xe2x80x9cLearned Classification of Sonar Targets Using a Massively Parallel Networkxe2x80x9d, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled xe2x80x9cProcess and Apparatus for the Automatic Detection and Extraction of Features in Images and Displaysxe2x80x9d to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled xe2x80x9cArtificial Neuron and Method of Using Same and U.S. Pat. No. 5,517,667 entitled xe2x80x9cNeural Network and Method of Using Samexe2x80x9d to S. T. Wang. Other examples include U.S. Pat. Nos. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744 (Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). All of the above references are included herein by reference.
4. Optics
Optics can be used in several configurations for monitoring the interior of a passenger compartment of an automobile. In one known method, a laser optical system uses a GaAs infrared laser beam to momentarily illuminate an object, occupant or child seat, in the manner as described and illustrated in FIG. 8 of U.S. Pat. No. 5,829,782 referenced above. The receiver can be a charge-coupled device or CCD (a type of TV camera), to receive the reflected light. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest within the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in the commonly owned ""462 as also described below.
In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to illuminate the desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. Recently, however, the cost of CCD and CMOS arrays has dropped substantially with the result that this configuration may now be the most cost-effective system for monitoring the passenger compartment as long as the distance from the transmitter to the objects is not needed. If this distance is required, then the laser system, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein is required. Alternately, a modulation system such as used with the laser distance system can be used with a CCD or CMOS camera and distance determined on a pixel by pixel basis.
A mechanical focusing system, such as used on some camera systems can determine the initial position of an occupant but is too slow to monitor his/her position during a crash. A distance measuring system based on focusing is described in U.S. Pat. No. 5,193,124 (Subbarao) which can either be used with a mechanical focusing system or with two cameras, the latter of which would be fast enough. Although the Subbarao patent provides a good discussion of the camera focusing art and is therefore incorporated herein by reference, it is a more complicated system than is needed for practicing the instant invention. In fact, a neural network can also be trained to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD""s and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes of the instant invention. Distance can also be determined by the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by the spreading or defocusing of a pattern of structured light projected onto the object of interest. Distance can also be measured by using time of flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching of this invention.
In each of these cases, regardless of the distance measurement system used, a trained pattern recognition system, as defined above, is used in the instant invention to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
5. Optics and Acoustics
The laser systems described above are expensive due to the requirement that they be modulated at a high frequency if the distance from the airbag to the occupant, for example, needs to be measured. Alternately, modulation of another light source such as an LED can be done and the distance measurement accomplished using a CCD or CMOS array on a pixel by pixel basis.
Both laser and non-laser optical systems in general are good at determining the location of objects within the two dimensional plane of the image and a pulsed laser radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time of flight through range gating techniques. It is also possible to determine distance with the non-laser system by focusing as discussed above, or stereographically if two spaced apart receivers are used and, in some cases the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements as discussed below.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z axis is from the transducer to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. Nos. 5,845,000 and 5,835,613.
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and CCD or CMOS array which receives and displays the reflected light and an analog to digital converter (ADC), or frame grabber, which digitizes the output of the CCD or CMOS and feeds it to an Artificial Neural Network (ANN) or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the driver""s ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant""s ears.
Although the use of ultrasound for distance measurement has many advantages, it also has some drawbacks. First, the speed of sound limits the rate at which the position of the occupant can be updated to approximately 10 milliseconds, which though sufficient for most cases, is marginal if the position of the occupant is to be tracked during a vehicle crash. Second, ultrasound waves are diffracted by changes in air density that can occur when the heater or air conditioner is operated or when there is a high-speed flow of air past the transducer. Third, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, the resolution of ultrasound is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
6. Applications
The applications for this technology are numerous as described in the patents and patent applications listed above. They include: (i) the monitoring of the occupant for safety purposes to prevent airbag deployment induced injuries, (ii) the locating of the eyes of the occupant (driver) to permit automatic adjustment of the rear view mirror(s), (iii) the location of the seat to place the occupant""s eyes at the proper position to eliminate the parallax in a heads-up display in night vision systems, (iv) the location of the ears of the occupant for optimum adjustment of the entertainment system, (v) the identification of the occupant for security reasons, (vi) the determination of obstructions in the path of a closing door or window, (vii) the determination of the position of the occupant""s shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant, (viii) the determination of the position of the rear of the occupants head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts. (ix) anticipatory crash sensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii) sunlight and headlight glare reduction and many others. In fact, over forty products alone have been identified based on the ability to identify and monitor objects and parts thereof in the passenger compartment of an automobile or truck.
7. Other Prior Art
European Patent Application No. 98110617.2 (Publication No. 0 885 782 A1), corresponding to U.S. patent application Ser. No. 08/872,836 filed Jun. 11, 1997, describes a purportedly novel motor vehicle control system including a pair of cameras which operatively produce first and second images of a passenger area. A distance processor determines the distances that a plurality of features in the first and second images are from the cameras based on the amount that each feature is shifted between the first and second images. An analyzer processes the determined distances and determines the size of an object on the seat. Additional analysis of the distance also may determine movement of the object and the rate of movement. The distance information also can be used to recognize predefined patterns in the images and this identify objects. An air bag controller utilizes the determined object characteristics in controlling deployment of the air bag.
A paper entitled xe2x80x9cSensing Automobile Occupant Position with Optical Triangulationxe2x80x9d by W. Chappelle, Sensors, December 1995, describes the use of optical triangulation techniques for determining the presence and position of people or rear-facing infant seats in the passenger compartment of a vehicle in order to guarantee the safe deployment of an air bag. The paper describes a system called the xe2x80x9cTakata Safety Shieldxe2x80x9d which purportedly makes high-speed distance measurements from the point of air bag deployment using a modulated infrared beam projected from an LED source. Two detectors are provided, each consisting of an imaging lens and a position-sensing detector.
A paper entitled xe2x80x9cAn Interior Compartment Protection System based on Motion Detection Using CMOS Imagersxe2x80x9d by S. B. Park et al., 1998 IEEE International Conference on Intelligent Vehicles, describes a purportedly novel image processing system based on a CMOS image sensor installed at the car roof for interior compartment monitoring including theft prevention and object recognition. One disclosed camera system is based on a CMOS image sensor and a near infrared (NIR) light emitting diode (LED) array.
A paper entitled xe2x80x9cA 256xc3x97256 CMOS Brightness Adaptive Imaging Array with Column-Parallel Digital Outputxe2x80x9d by C. Sodini et al., 1988 IEEE International Conference on Intelligent Vehicles, describes a CMOS image sensor for intelligent transportation system applications such as adaptive cruise control and traffic monitoring. Among the purported novelties is the use of a technique for increasing the dynamic range in a CMOS imager by a factor of approximately 20, which technique is based on a previously described technique for CCD imagers.
A paper entitled xe2x80x9cIntelligent System for Video Monitoring of Vehicle Cockpitxe2x80x9d by S. Boveric et al., SAE Technical Paper Series No. 980613, Feb. 23-26, 1998, describes the installation of an optical/retina sensor in the vehicle and several uses of this sensor. Possible uses are said to include observation of the driver""s face (eyelid movement) and the driver""s attitude to allow analysis of the driver""s vigilance level and warn him/her about critical situations and observation of the front passenger seat to allow the determination of the presence of somebody or something located on the seat and to value the volumetric occupancy of the passenger for the purpose of optimizing the operating conditions for air bags.
Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an image analyzer (MS in FIG. 1) for analyzing the position of driver including an infrared light source which illuminates the driver""s face and an image detector which receives light from the driver""s face, determines the position of facial feature, e.g., the eyes in three dimensions, and thus determines the position of the driver in three dimensions. A pattern recognition process is used to determine the position of the facial features and entails converting the pixels forming the image to either black or white based on intensity and conducting an analysis based on the white area in order to find the largest contiguous white area and the center point thereof. Based on the location of the center point of the largest contiguous white area, the driver""s height is derived and a heads up display is adjusted so information is within driver""s field of view. The pattern recognition process can be applied to detect the eyes, mouth, or nose of the driver based on the differentiation between the white and black areas.
Ando (U.S. Pat. No. 5,008,946) describes a system which recognizes an image and specifically ascertains the position of the pupils and mouth of the occupant to enable movement of the pupils and mouth to control electrical devices installed in the automobile. The system includes a camera which takes a picture of the occupant and applies algorithms based on pattern recognition techniques to analyze the picture, converted into an electrical signal, to determine the position of certain portions of the image, namely the pupils and mouth.
Masamori (U.S. Pat. No. 5,227,784) describes a system which is based on radar, specifically it is a collision avoidance system aimed at detecting vehicles which are at some distance from the vehicle.
Suzuki et al. (U.S. Pat. No. 5,026,153) describes a vehicle tracking control for continuously detecting the distance and direction to a preceding vehicle irrespective of background dark/light distribution. In this system, even vehicle must have a light on its rear that emits a constant or time varying signal and two photoelectric sensors that zero in on the light emitted from the preceding vehicle are used and thereby determine both the distance and angular position of the preceding vehicle.
Krumm (U.S. Pat. No. 5,983.147) describes a system for determining the occupancy of a passenger compartment including a pair of cameras mounted so as to obtain binocular stereo images of the same location in the passenger compartment. A representation of the output from the cameras is compared to stored representations of known occupants and occupancy situations to determine which stored representation the output from the cameras most closely approximates. The stored representations include that of the presence or absence of a person or an infant seat in the front passenger seat.
Farmer et al. (U.S. Pat. No. 6,005,958) describes a method and system for detecting the type and position of a vehicle occupant utilizing a single camera unit. The single camera unit is positioned at the driver or passenger side A-pillar in order to generate data of the front seating area of the vehicle. The type and position of the occupant is used to optimize the efficiency and safety in controlling deployment of an occupant protection device such as an air bag.
A paper by Rudolph Schwarte, et al. entitled xe2x80x9cNew Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technologyxe2x80x9d. Eds. S. Krueger, W. Gessner, Proceedings of the AMAA 2000 Advanced Microsystems for Automotive Applications 2000. Springer Verlag; Berlin, Heidelberg, New York, ISBN 3-540670874, describes an implementation of the teachings of the instant invention wherein a modulated light source is used in conjunction with phase determination circuitry to locate the distance to objects in the image on a pixel by pixel basis. This camera is an active pixel camera the use of which for internal and external vehicle monitoring is also a teaching of this invention The novel feature of the PMD camera is that the pixels are designed to provide a distance measuring capability within each pixel itself. This then is a novel application of the active pixel and distance measuring teachings of the instant invention
The instant invention as described in the above-referenced commonly assigned patents and patent applications, teaches the use of modulating the light used to illuminate an object and to determine the distance to that object based on the phase difference between the reflected radiation and the transmitted radiation. The illumination can be modulated at a single frequency when short distances such as within the passenger compartment are to be measured. Typically, the modulation wavelength would be selected such that one wave would have a length of approximately one meter or less. This would provide resolution of I cm or less. For larger vehicles, a longer wavelength would be desirable. For measuring longer distances, the illumination can be modulated at more than one frequency to eliminate cycle ambiguity if there is more than one cycle between the source of illumination and the illuminated object. This technique is particularly desirable when monitoring objects exterior to the vehicle to permit accurate measurements of devices that are hundreds of meters from the vehicle as well as those that are a few meters away.
Although a simple frequency modulation scheme has been disclosed so far, it is also possible to use other coding techniques including the coding of the illumination with one of a variety of correlation patterns including a pseudo-random code. Similarly, although frequency and code domain systems have been described, time domain systems are also applicable wherein a pulse of light is emitted and the time of flight measured. Additionally, in the frequency domain case, a chirp can be emitted and the reflected light compared in frequency with the chirp to determine by frequency difference the distance to the object. Although each of these techniques is known to those skilled in the art, they have heretofore not been applied for monitoring objects within or outside of a vehicle.
Principle objects and advantages of the optical sensing system in accordance with the invention are.
1. To recognize the presence of a human on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag, heating and air conditioning, or entertainment systems, among others.
2. To recognize the presence of a human on a particular seat of a motor vehicle and then to determine his/her position and to use this position information to affect the operation of another vehicle system.
3. To determine the position, velocity or size of an occupant in a motor vehicle and to utilize this information to control the rate of gas generation, or the amount of gas generated by an airbag inflator system.
4. To determine the presence or position of rear seated occupants in the vehicle and to use this information to affect the operation of a rear seat protection airbag for frontal, side and/or rear impacts.
5. To recognize the presence of a rear facing child seat on a particular seat of a motor vehicle and to use this information to affect the operation of another vehicle system such as the airbag system.
6. To determine the approximate location of the eyes of a driver and to use that information to control the position of one or more of the rear view mirrors of the vehicle.
7. To monitor the position of the head of the vehicle driver and determine whether the driver is falling asleep or otherwise impaired and likely to lose control of the vehicle and to use that information to affect another vehicle system.
8. To provide an occupant position sensor which reliably permits, and in a timely manner, a determination to be made that the occupant is out-of-position, or will become out-of-position, and likely to be injured by a deploying airbag and to then output a signal to suppress the deployment of the airbag.
9. To provide an anticipatory sensor that permits accurate identification of the about-to-impact object in the presence of snow and/or fog whereby the sensor is located within the vehicle.
10. To provide a smart headlight dimmer system which senses the headlights from an oncoming vehicle or the tail lights of a vehicle in front of the subject vehicle and identifies these lights differentiating them from reflections from signs or the road surface and then sends a signal to dim the headlights.
11. To provide a blind spot detector which detects and categorizes an object in the driver""s blind spot or other location in the vicinity of the vehicle and warns the driver in the event the driver begins to change lanes, for example, or continuously informs the driver of the state of occupancy of the blind spot.
12. To provide a occupant position determination in a sufficiently short time that the position of an occupant can be tracked during a vehicle crash.
13. To provide an occupant vehicle interior monitoring system which is not affected by temperature or thermal gradients.
14. To provide an occupant vehicle interior monitoring system which has high resolution to improve system accuracy and permits the location of body parts of the occupant to be determined.
15. To provide an occupant vehicle interior monitoring system which reduces the glare from sunlight and headlights by imposing a filter between the eyes of an occupant and the light source.
16. To provide a camera system for interior and exterior monitoring, which can adjust on a pixel by, pixel basis for the intensity of the received light.
17. To provide for the use of an active pixel camera for interior and exterior vehicle monitoring.
18. To provide a system for recognizing the identity of a particular individual in the vehicle.
19. To use the principles of time of flight to measure the distance to an occupant or object exterior to the vehicle.
20. To obtain a three dimensional image from a device at one location on a vehicle.
21. To use pattern recognition techniques for analyzing three-dimensional image data of occupants of a vehicle and objects exterior to the vehicle.
22. To provide a system of frequency domain modulation of the illumination of an object interior or exterior of a vehicle.
23. To utilize an active pixel camera for monitor objects interior or exterior to the vehicle.
24. To use a chirp frequency modulation technique to aid in determining the distance to an object interior or exterior of a vehicle.
25. To utilize a correlation pattern modulation in a form of code division modulation for determining the distance of an object interior or exterior of a vehicle.
These and other objects and advantages will become apparent from the following description of the preferred embodiments of the vehicle identification and monitoring system of this invention.
Briefly, in order to achieve at least one of the objects, a system for controlling operation of a vehicle based on recognition of an authorized individual comprises a processor embodying a pattern recognition algorithm, as defined above, trained to identify whether a person is the individual by analyzing data derived from images and one or more optical receiving units for receiving an optical image including the person and deriving data from the image. Each optical receiving unit is coupled to the processor to provide the data to the pattern recognition algorithm to thereby obtain an indication from the pattern recognition algorithm whether the person is the individual. A security system is arranged to enable operation of the vehicle when the pattern recognition algorithm provides an indication that the person is an individual authorized to operate the vehicle and prevent operation of the vehicle when the pattern recognition algorithm does not provide an indication that the person is an individual authorized to operate the vehicle.
An optional optical transmitting unit is provided in the vehicle for transmitting electromagnetic energy and is arranged relative to the optical receiving unit(s) such that electromagnetic energy transmitted by the optical transmitting unit is reflected by the person and received by at least one of the optical receiving units.
The optical receiving units may be selected from a group consisting of a CCD array, a CMOS array, an active pixel camera and an HDRC camera. Other types of two or three-dimensional cameras can also be used.
Instead of a security system, the individual recognition system can be used to control vehicular components, such as the mirrors, the seat, the anchorage point of the seatbelt, the airbag deployment parameters including inflation rate and pressure, inflation direction, deflation rate, time of inflation, the headrest, the steering wheel, the pedals, the entertainment system and the airconditioning/ventilation system. In this case, the system includes control means coupled to the component for affecting the component based on the indication from the pattern recognition algorithm whether the person is the individual.
A method for controlling operation of a vehicle based on recognition of a person as one of a set of authorized individuals comprises the steps of obtaining images including the authorized individuals by means of one or more optical receiving unit, deriving data from the images, training a pattern recognition algorithm on the data derived from the images which is capable of identifying a person as one of the individuals, then subsequently obtaining images by means of the optical receiving unit(s), inputting data derived from the images subsequently obtained by the optical receiving unit(s) into the pattern recognition algorithm to obtain an indication whether the person is one of the set of authorized individuals, and providing a security system which enables operation of the vehicle when the pattern recognition algorithm provides an indication that the person is one of the set of individuals authorized to operate the vehicle and prevents operation of the vehicle when the pattern recognition algorithm does not provide an indication that the person is one of the set of individuals authorized to operate the vehicle.
The data derivation from the images may entail any number of image processing techniques including eliminating pixels from the images which are present in multiple images and comparing the images with stored arrays of pixels and eliminating pixels from the images which are present in the stored arrays of pixels.
The method can also be used to control a vehicular component based on recognition of a person as one of a predetermined set of particular individuals. This method includes the step of affecting the component based on the indication from the pattern recognition algorithm whether the person is one of the set of individuals. The components may be one or more of the following: the mirrors, the seat, the anchorage point of the seatbelt, the airbag deployment parameters including inflation rate and pressure, inflation direction, deflation rate, time of inflation, the headrest, the steering wheel, the pedals, the entertainment system and the air-conditioning/ventilation system.